fix(detector): cap decompressed content-stream size - #418
Open
abimaelmartell wants to merge 1 commit into
Open
Conversation
Stop holding the full inflated page or Form stream in the detector so a highly compressible Flate stream cannot balloon resident memory. Co-authored-by: Cursor <cursoragent@cursor.com>
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
2 issues found across 4 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/stream_decode.rs">
<violation number="1" location="src/stream_decode.rs:41">
P2: When a Flate content stream has PNG predictor parameters, this branch scans predictor-encoded bytes instead of applying predictor reversal. Apply predictor decoding incrementally or route predictor streams through a correctly bounded decoder, or text pages using them can be misclassified.</violation>
<violation number="2" location="src/stream_decode.rs:48">
P1: When a content stream uses a filter chain, this branch materializes the entire decoded buffer before checking its size. Decode every supported filter incrementally or reject the stream before this call, otherwise a small multi-filter stream bypasses the 32 MiB protection.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Fix all with cubic | Re-trigger cubic
| if stream.content.len() > max_bytes { | ||
| return None; | ||
| } | ||
| match stream.decompressed_content() { |
There was a problem hiding this comment.
P1: When a content stream uses a filter chain, this branch materializes the entire decoded buffer before checking its size. Decode every supported filter incrementally or reject the stream before this call, otherwise a small multi-filter stream bypasses the 32 MiB protection.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/stream_decode.rs, line 48:
<comment>When a content stream uses a filter chain, this branch materializes the entire decoded buffer before checking its size. Decode every supported filter incrementally or reject the stream before this call, otherwise a small multi-filter stream bypasses the 32 MiB protection.</comment>
<file context>
@@ -0,0 +1,145 @@
+ if stream.content.len() > max_bytes {
+ return None;
+ }
+ match stream.decompressed_content() {
+ Ok(data) if data.len() <= max_bytes => Some(data),
+ Ok(_) => None,
</file context>
| // Plain Flate is the highly compressible case. Detector scans only need | ||
| // the inflated operator bytes; skip PNG predictors here so inflate can | ||
| // stop at the budget instead of materializing the full buffer first. | ||
| if filters.len() == 1 && filters[0] == b"FlateDecode" { |
There was a problem hiding this comment.
P2: When a Flate content stream has PNG predictor parameters, this branch scans predictor-encoded bytes instead of applying predictor reversal. Apply predictor decoding incrementally or route predictor streams through a correctly bounded decoder, or text pages using them can be misclassified.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/stream_decode.rs, line 41:
<comment>When a Flate content stream has PNG predictor parameters, this branch scans predictor-encoded bytes instead of applying predictor reversal. Apply predictor decoding incrementally or route predictor streams through a correctly bounded decoder, or text pages using them can be misclassified.</comment>
<file context>
@@ -0,0 +1,145 @@
+ // Plain Flate is the highly compressible case. Detector scans only need
+ // the inflated operator bytes; skip PNG predictors here so inflate can
+ // stop at the budget instead of materializing the full buffer first.
+ if filters.len() == 1 && filters[0] == b"FlateDecode" {
+ return inflate_flate_bounded(&stream.content, max_bytes);
+ }
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
cargo test --lib stream_decodecargo test --lib detector::tests::flate_page_content_still_finds_text_operators detector::tests::flate_form_xobject_still_finds_text_operatorsdetect-pdfon a normal text PDF still reports extractable textMade with Cursor
Summary by cubic
Caps decompressed PDF content-stream size during detector scans at 32 MiB to prevent highly compressible
FlateDecodestreams from ballooning memory. Previously the detector fully decompressed page and Form streams; now it inflates incrementally and skips any stream that would exceed the cap.stream_decode.rswith bounded inflate forFlateDecode; other filters fall back to full decode only if within budget.stream_content_for_scanin page and Form XObject scans; over-budget streams return empty content and are skipped for that scan.flate2 = "1.1".Written for commit 120f7f5. Summary will update on new commits.